Testing Setfos

Why testing?

Murphy’s law

Anything that can go wrong will go wrong

(and at the worst possible time)

Corollaries

Any feature that is not tested is broken.

Any feature that is tested manually is broken the moment changes are made

(anywhere in the code)

Automated testing

Challenges I: Comprehensive tests

Test legacy code for inverse sqrt

float Q_rsqrt( float number )
{
 long i; float x2, y; const float threehalfs = 1.5F;
 x2 = number * 0.5F;
 y  = number;
 // evil floating point bit level hacking
 i  = * ( long * ) &y;
// wtf?
 i  = 0x5f3759df - ( i >> 1 );
 y  = * ( float * ) &i;
 y  = y * ( threehalfs - ( x2 * y * y ) );   // 1st iteration
// y  = y * ( threehalfs - ( x2 * y * y ) );   // 2nd iteration
 return y;
}

Comprehensive tests - continued

  • Testing all values: not feasible
  • Equivalence classes and their boundaries
  • Problem dependent
  • Implementation dependent

Challenges II: Combinations of parameters

  • Most problems come from combination of not more than two parameters
  • Don’t test all combinations
  • Test all pairs of combinations

Challenges III: sensitivity vs. specificity

  • Sensitivity: if something is broken at least one test complains
  • Specificity: if something is broken only a few tests complain
  • Usually contradictory goals

Types of tests

Unit test

Unit test: Testing individual units (e.g. inverse sqrt method)

  • Highly specific
  • Easy to write
  • Units not isolated
  • Does not test integration

134 unit tests in setfos kernel + GUI tests

Integration tests

  • In setfos: run simulations and compare to reference results
  • ~730 integration tests
  • Highly sensitive
  • Not very specific

End-to-end tests/manual tests

  • E2E tests very hard to automate
  • In setfos, all are done manually
  • Installation tests/run individual simulations for each hotfix
  • Internal customers

GUI

  • Architecture unit tests: test dependency of components (workspace, results, etc.)
  • Parser + serialization tests
  • Kernel + GUI interaction tests: GUI loads models and starts simulation and compares results
  • No automated E2E tests

Automated testing